Papers with Uncertainty Estimation
MARS: Meaning-Aware Response Scoring for Uncertainty Estimation in Generative LLMs (2024.acl-long)
Copied to clipboard
Yavuz Faruk Bakman, Duygu Nur Yaldiz, Baturalp Buyukates, Chenyang Tao, Dimitrios Dimitriadis, Salman Avestimehr
| Challenge: | Generative Large Language Models (LLMs) are widely utilized for their excellence in various tasks. however, their tendency to produce inaccurate or misleading outputs poses a potential risk. |
| Approach: | They propose a new scoring function that considers the semantic contribution of each token in the generated sequence in the context of the question. |
| Outcome: | The proposed scoring function improves UE performance on a medical QA dataset. |
Why Uncertainty Estimation Methods Fall Short in RAG: An Axiomatic Analysis (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing UE methods cannot reliably estimate the correctness of LLM responses in Retrieval-Augmented Generation (RAG) . Existing methods generate low uncertainty values without considering relevance of context to query . |
| Approach: | They propose an axiomatic framework to identify deficiencies in existing UE methods and introduce five constraints that an effective UE method should meet after incorporating retrieved documents into the LLM’s prompt. |
| Outcome: | The proposed framework satisfies all the axioms and improves correlation between uncertainty estimates and correctness. |
Towards Harmonized Uncertainty Estimation for Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated exceptional capabilities in handling a wide range of downstream tasks. |
| Approach: | They propose a method that employs a lightweight model trained on data aligned with the target LLM’s performance to adjust uncertainty scores. |
| Outcome: | The proposed method achieves improvements of up to 60% over existing methods. |